Calibration device, calibration method, and computer readable medium for visual sensor

ABSTRACT

For calibration on a single camera or a stereo camera, a calibration range is set in advance in an image coordinate system and the calibration is performed in an arbitrary range. A visual sensor controller is a calibration device that associates a robot coordinate system at a robot and an image coordinate system at a camera by placing a target mark at the robot, moving the robot, and detecting the target mark at multiple points in a view of the camera. The calibration device comprises: an image range setting unit that sets an image range in the image coordinate system at the camera; and a calibration range measurement unit that measures an operation range for the robot corresponding to the image range before implementation of calibration by moving the robot and detecting the target mark.

This application is based on and claims the benefit of priority fromJapanese Patent Application No. 2017-003667, filed on 12 Jan. 2017, thecontent of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to calibration on a visual sensor,particularly, to a calibration device, a calibration method, andcomputer readable medium for measuring a range in which a target mark isdetected during implementation of calibration using a stereo camera withmultiple cameras.

Related Art

In a robot system, a robot is given a visual function. Operation such ashandling or machining on a work is done by making the robot recognizethe position information of a subject. This visual function is fulfilledby capturing an image of the subject with a visual sensor attached to ahand or a neighboring part of the robot or a visual sensor providedaround the robot. In such a robot system, to acquire the positioninformation of the subject viewed from the robot, calibration data isrequired for converting the position information of the subject in animage to the position information of the subject viewed from the robot.

Calibration data has been acquired by various conventional methods. Forexample, patent document 1 suggests a method of attaching a latticepattern to the end of a robot arm and measuring the pattern with afixedly arranged visual sensor (this method will be called a “methodA”). Patent document 2 suggests a method of performing calibration byattaching a target mark having a position and a posture determined inadvance in an end point coordinate system at a robot to the end of anarm, and determining the position of the target mark at multiple pointsin an image of the target mark captured by a visual sensor (this methodwill be called a “method B”).

If calibration is to be performed by the method A, a pattern to be usedfor the calibration should be prepared in advance. If the view of acamera is too wide or too narrow for the prepared pattern, the method Afails to perform high-precision calibration. In contrast, calibration bythe method B allows calibration in a wider view or calibration in anarrower view than in the method A, thereby advantageously increasingthe degree of freedom of the calibration. For three-dimensionalmeasurement, a stereo camera has been used in some cases as a visualsensor as described in patent document 3, for example. There have beenthe following systems for the stereo camera: a passive stereo system ofmatching corresponding points by using the texture of a subject; and anactive stereo system of matching corresponding points by using a patternprojected on a subject. In either case, calibration is required on twoor more cameras forming the stereo camera.

-   Patent Document 1: Japanese Patent No. 2690603-   Patent Document 2: Japanese Unexamined Patent Application,    Publication No. 2015-174191-   Patent Document 3: Japanese Unexamined Patent Application,    Publication No. 2010-172986

SUMMARY OF THE INVENTION

To calibrate a visual sensor (camera, for example) more precisely, atarget mark is desirably moved through a wide area in a view range whilethe visual sensor (camera, for example) is calibrated. If the targetmark is to collide with an obstacle while being moved in the view range,for example, a range for move of the target mark should be limited.Hence, before the visual sensor (camera, for example) is calibrated, arange of move of the target mark (hereinafter also called a “calibrationrange”) for capturing an image of the target mark having been moved isrequired to be set in some way in a plane on which the target mark is tobe moved during implementation of the calibration so as to allow thetarget mark to be moved through a wide area in the view range. Thecalibration range can also be designated in a robot coordinate system.However, to designate a range in the robot coordinate system for makingthe target mark come into the view and preventing the target mark fromcolliding with an obstacle, a user is required to check the target markin an image of the target mark captured by the camera while operating arobot by hand. To eliminate the need for such troublesome operation, thecalibration range is desirably set in a captured image. For calibrationon multiple cameras such as those of a stereo camera, the multiplecameras are attached at separate positions, so that each camera isrequired to be calibrated independently. Hence, also in this case,before each camera is calibrated, a range of move of the target mark(calibration range) is required to be set for each of a first camera 21and a second camera 22 in a plane on which the target mark is to bemoved in such a manner that an image of the target mark having beenmoved can be captured in a range of image capture (angle of view) byeach camera.

In this regard, patent document 2 does not recite that a range of moveof a work 1 (calibration range) is set on a plane on which the work 1 isto move and in a range of image capture (angle of view) by an imagecapture unit 40 before the image capture unit 40 is calibrated. Further,the calibration described in patent document 2 is not to calibratemultiple cameras such as those of a stereo camera but is merely tocalibrate a single camera. The calibration described in patent document3 is performed by attaching a checker board as a basic matrixcalculation tool to the end of a robot arm and capturing images of thechecker board with a stereo camera. Thus, this calibration inherentlydoes not correspond to calibration performed by attaching a target markto the end of an arm, and determining the position of the target mark atmultiple points where the target mark has been moved in the images ofthe target mark captured by the stereo camera.

The present invention provides a calibration device, a calibrationmethod, and a program capable of setting a range of move of a targetmark (calibration range) in advance in space for move of the target markso as to make the range of move fall in a view range for a single visualsensor (camera, for example) or in a view range for each camera forminga stereo camera.

(1) A calibration device according to the present invention (“visualsensor controller 1” described later, for example) is a calibrationdevice that associates a robot coordinate system at a robot (“robot 4”described later, for example) and an image coordinate system at a camera(“camera 2” described later, for example) by placing a target mark(“target mark 5” described later, for example) at the robot, controllingthe robot so as to move the target mark, and detecting the target markat multiple points in a view of the camera. The calibration devicecomprises: an image range setting unit (“first image range setting unit105” described later, for example) that sets an image range in the imagecoordinate system; and a calibration range measurement unit (“firstcalibration range measurement unit 106” described later, for example)that measures a calibration range as an operation range for the robotcorresponding to the image range before implementation of calibration bycontrolling the robot to move the target mark and detecting the targetmark. During implementation of the calibration, the robot is controlledso as to move the target mark in the calibration range.

(2) In the calibration device described in (1), the calibration rangemay be configured to be set on a plane.

(3) In the calibration device described in (1) or (2), the calibrationrange may be configured to be set on a plane vertical to an optical axisof the camera.

(4) In the calibration device described in (1) or (2), the calibrationrange may be configured to be set on a plane tilted from an optical axisof the camera.

(5) In the calibration device described in (1) or (4), the calibrationrange may be configured to be set on each of at least two planes.

(6) A calibration method according to the present invention (“visualsensor control method” described later, for example) is a calibrationmethod implemented by a calibration device that associates a robotcoordinate system at a robot (“robot 4” described later, for example)and an image coordinate system at a camera (“camera 2” described later,for example) by placing a target mark (“target mark 5” described later,for example) at the robot, controlling the robot so as to move thetarget mark, and detecting the target mark at multiple points in a viewof the camera. The calibration method comprises: an image range settingstep (“first image range setting step” described later, for example) ofsetting an image range in the image coordinate system; and a calibrationrange measurement step (“calibration range measurement step” describedlater, for example) of measuring a calibration range as an operationrange for the robot corresponding to the image range beforeimplementation of calibration by controlling the robot to move thetarget mark and detecting the target mark. During implementation of thecalibration, the robot is controlled so as to move the target mark inthe calibration range.

(7) A program according to the present invention (“program” describedlater, for example) causes a computer to execute the following steps.The computer controls a calibration device that associates a robotcoordinate system at a robot (“robot 4” described later, for example)and an image coordinate system at a camera (“camera 2” described later,for example) by placing a target mark (“target mark 5” described later,for example) at the robot, controlling the robot so as to move thetarget mark, and detecting the target mark at multiple points in a viewof the camera. The steps comprise: an image range setting step (“firstimage range setting step” described later, for example) of setting animage range in the image coordinate system; and a calibration rangemeasurement step (“calibration range measurement step” described later,for example) of measuring a calibration range as an operation range forthe robot corresponding to the image range before implementation ofcalibration by controlling the robot to move the target mark anddetecting the target mark. During implementation of the calibration, therobot is controlled so as to move the target mark in the calibrationrange.

(8) A calibration device according to the present invention (“visualsensor controller 1A” described later, for example) is a calibrationdevice that associates a robot coordinate system at a robot (“robot 4”described later, for example), position information in an imagecoordinate system at a first camera (“first camera 21” described later,for example) of a stereo camera (“stereo camera 2A” described later, forexample), and position information in an image coordinate system at asecond camera (“second camera 22” described later, for example) of thestereo camera by placing a target mark (“target mark 5” described later,for example) at the robot, controlling the robot so as to move thetarget mark, and detecting the target mark at multiple points in a viewof the stereo camera including at least the first camera and the secondcamera. The calibration device comprises: a first image range settingunit (“first image range setting unit 105” described later, for example)that sets a first image range in the image coordinate system at thefirst camera; a second image range setting unit (“second image rangesetting unit 1052” described later, for example) that sets a secondimage range in the image coordinate system at the second camera; a firstcalibration range measurement unit (“first calibration range measurementunit 106” described later, for example) that measures a firstcalibration range as a first operation range for the robot correspondingto the first image range before the first camera and the second cameraare calibrated by controlling the robot to move the target mark anddetecting the target mark by using the first camera; and a secondcalibration range measurement unit (“second calibration rangemeasurement unit 1062” described later, for example) that measures asecond calibration range as a second operation range for the robotcorresponding to the second image range before the first camera and thesecond camera are calibrated by controlling the robot to move the targetmark and detecting the target mark by using the second camera. While thefirst camera and the second camera are calibrated, the robot iscontrolled so as to move the target mark in at least one of the firstcalibration range and the second calibration range, or in a rangecovered by both the first calibration range and the second calibrationrange.

(9) In the calibration device described in (8), each of the firstcalibration range and the second calibration range may be configured tobe set on a plane.

(10) A calibration method according to the present invention (“visualsensor control method” described later, for example) is a calibrationmethod implemented by a calibration device that associates a robotcoordinate system at a robot (“robot 4” described later, for example),position information in an image coordinate system at a first camera(“first camera 21” described later, for example) of a stereo camera(“stereo camera 2A” described later, for example), and positioninformation in an image coordinate system at a second camera (“secondcamera 22” described later, for example) of the stereo camera by placinga target mark (“target mark 5” described later, for example) at therobot, controlling the robot so as to move the target mark, anddetecting the target mark at multiple points in a view of the stereocamera including at least the first camera and the second camera. Thecalibration method comprises: a first image range setting step (“firstimage range setting step” described later, for example) of setting afirst image range in the image coordinate system at the first camera; asecond image range setting step (“second image range setting step”described later, for example) of setting a second image range in theimage coordinate system at the second camera; a first calibration rangemeasurement step (“first calibration range measurement step” describedlater, for example) of measuring a first calibration range as a firstoperation range for the robot corresponding to the first image rangebefore the first camera and the second camera are calibrated bycontrolling the robot to move the target mark and detecting the targetmark by using the first camera; and a second calibration rangemeasurement step (“second calibration range measurement step” describedlater, for example) of measuring a second calibration range as a secondoperation range for the robot corresponding to the second image rangebefore the first camera and the second camera are calibrated bycontrolling the robot to move the target mark and detecting the targetmark by using the second camera. While the first camera and the secondcamera are calibrated, the robot is controlled so as to move the targetmark in at least one of the first calibration range and the secondcalibration range, or in a range covered by both the first calibrationrange and the second calibration range.

(11) A program according to the present invention (“program” describedlater, for example) causes a computer to execute the following steps.The computer controls a calibration device that associates a robotcoordinate system at a robot (“robot 4” described later, for example),position information in an image coordinate system at a first camera(“first camera 21” described later, for example) of a stereo camera(“stereo camera 2A” described later, for example), and positioninformation in an image coordinate system at a second camera (“secondcamera 22” described later, for example) of the stereo camera by placinga target mark (“target mark 5” described later, for example) at therobot, controlling the robot so as to move the target mark, anddetecting the target mark at multiple points in a view of the stereocamera including at least the first camera and the second camera. Thesteps comprise: a first image range setting step (“first image rangesetting step” described later, for example) of setting a first imagerange in the image coordinate system at the first camera; a second imagerange setting step (“second image range setting step” described later,for example) of setting a second image range in the image coordinatesystem at the second camera; a first calibration range measurement step(“first calibration range measurement step” described later, forexample) of measuring a first calibration range as a first operationrange for the robot corresponding to the first image range before thefirst camera and the second camera are calibrated by controlling therobot to move the target mark and detecting the target mark by using thefirst camera; and a second calibration range measurement step (“secondcalibration range measurement step” described later, for example) ofmeasuring a second calibration range as a second operation range for therobot corresponding to the second image range before the first cameraand the second camera are calibrated by controlling the robot to movethe target mark and detecting the target mark by using the secondcamera. While the first camera and the second camera are calibrated, therobot is controlled so as to move the target mark in at least one of thefirst calibration range and the second calibration range, or in a rangecovered by both the first calibration range and the second calibrationrange.

A calibration device, a calibration method, and a program provided bythe present invention are capable of performing calibration with anincreased degree of freedom by measuring an operation range for a robot(calibration range) corresponding to a range of move of a target mark inadvance before implementation of the calibration in space where thetarget mark is to be moved during implementation of the calibration,based on a view range for a single visual sensor or a view range foreach camera forming a stereo camera.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the configuration of a robot system 1000 entirely;

FIG. 2 shows examples of a target mark 5;

FIG. 3 is a functional block diagram showing the functionalconfiguration of a visual sensor controller 1;

FIG. 4 is a block diagram showing the functional configuration of a CPU10 in the visual sensor controller 1;

FIG. 5 shows a flowchart for generating a model pattern;

FIG. 6 shows an example of a model pattern designation area;

FIG. 7 shows an example of a point P where the three-dimensionalposition information of the target mark 5 is to be measured;

FIG. 8A shows a flowchart showing a flow followed by a first calibrationrange measurement unit 106 for measuring a coordinate positioninformation in a robot coordinate system showing a boundary of acalibration range;

FIG. 8B shows a flowchart showing the flow followed by the firstcalibration range measurement unit 106 for measuring the coordinateposition information in the robot coordinate system showing the boundaryof the calibration range;

FIG. 9A shows an example of a path traced by the target mark 5 attachedto an end portion of an arm 41 while the target mark 5 is moved in acalibration range;

FIG. 9B shows an example of a path traced by the target mark 5 attachedto the end portion of the arm 41 while the target mark 5 is moved in thecalibration range;

FIG. 10 is a flowchart showing calibration process on a camera 2according to a first embodiment;

FIG. 11A shows an example of a plane on which a calibration range isset;

FIG. 11B shows examples of planes on which a calibration range is set;

FIG. 12 shows the configuration of a robot system 1000 entirely forperforming calibration by using a stereo camera 2A according to a secondembodiment;

FIG. 13A shows an example of arrangement of the stereo camera 2A;

FIG. 13B shows an example of arrangement of the stereo camera 2A;

FIG. 14 is a block diagram showing the functional configuration of a CPU10 in a visual sensor controller 1A; and

FIG. 15 shows a flowchart showing calibration process on the stereocamera 2A according to the second embodiment.

DETAILED DESCRIPTION OF THE INVENTION First Embodiment

An example of an embodiment (first embodiment) of the present inventionwill be described below. FIG. 1 shows the configuration of a robotsystem 1000 entirely for calibration on a visual sensor. The robotsystem 1000 is for performing calibration using a single camera 2. Asshown in FIG. 1, the robot system 1000 includes: a visual sensorcontroller 1 that makes three-dimensional measurement through imageprocessing on data about an image captured by the single camera 2; arobot 4 having an arm 41 with an end portion to which a target mark 5 isattached; and a robot controller 3 for control over the robot 4. Thecamera 2 is fixed to a pedestal (not shown in the drawings). In thisembodiment, the visual sensor controller 1 is described as an example ofa calibration device.

FIG. 2 shows examples of the target mark 5. The target mark 5 is notlimited to these examples. Any shape is applicable to the target mark 5.Meanwhile, the shape of the target mark 5 is desirably such that thecharacteristics of the target mark 5 used as a model pattern areexpressed on a two-dimensional plane. The target mark 5 may be a markprinted on paper or a seal and attached to the end of the arm 41 of therobot 4, for example.

The robot controller 3 recognizes the coordinate position information ofthe end of the arm 41 in a robot coordinate system as a currentposition. Thus, based on the coordinate position of the end of the arm41 in the robot coordinate system and the known three-dimensionalposition information and the known three-dimensional posture of thetarget mark 5 in an end point coordinate system at the robot 4, therobot controller 3 can always recognize the coordinate position of thetarget mark 5 in the robot coordinate system while the robot controller3 controls drive of the arm 41. The robot controller 3 includes a CPU(not shown in the drawings) for controlling the robot controller 3entirely in an integrated fashion. The visual sensor controller 1 isconnected to the robot controller 3 through an external equipmentinterface (not shown in the drawings). The robot controller 3 transmitsa signal for triggering image processing to the visual sensor controller1. Further, the robot controller 3 receives a result of image processingobtained by execution of the image processing (detection of the targetmark 5, for example) by the visual sensor controller 1, etc.

The robot controller 3 controls drive of the arm 41 so as to move thetarget mark 5 attached to the end of the arm 41 in a range set inadvance for calibration (“calibration range”) during implementation ofthe calibration. Process of measuring the calibration range will bedescribed in detail later.

The robot controller 3 measures the coordinate position of the targetmark 5 in the robot coordinate system attached to the end of the arm 41of the robot 4. Specifically, the robot controller 3 can measure thecoordinate position of the target mark 5 in the robot coordinate systemwhile the target mark 5 is at a destination.

[Visual Sensor Controller 1]

The camera 2 is connected to the visual sensor controller 1 functioningas the calibration device. The visual sensor controller 1 makes thecamera 2 capture an image of the target mark 5 to calibrate the camera2.

FIG. 3 is a functional block diagram showing the functionalconfiguration of the visual sensor controller 1. The visual sensorcontroller 1 includes a central processing unit (CPU) 10 for controllingthe visual sensor controller 1 entirely in an integrated fashion. TheCPU 10 is connected through a bus 11 to multiple frame memories 12, aread-only memory (ROM) 13, a random access memory (RAM) 14, and anonvolatile RAM 15. The camera 2 is connected to the bus 11 through acamera interface 16. Further, a monitor 19 is connected to the bus 11through a monitor interface 17. The CPU 10 is further connected throughthe bus 11 to an external equipment interface 18. The external equipmentinterface 18 is connected to the robot controller 3 to receive thecoordinate position of the target mark 5 from the robot controller 3 andtransmit a result of image processing obtained by execution of the imageprocessing (detection of the target mark 5, for example) by the visualsensor controller 1, etc. to the robot controller 3.

The ROM 13 stores programs used for execution of various types ofprocess by the visual sensor controller 1. Generally, access is madefaster to a RAM than to a ROM. Thus, the CPU 10 may develop the programsstored in the ROM 13 in advance on the RAM 14. Then, the CPU 10 may readthe programs from the RAM 14 and execute the read programs. The RAM 14stores temporarily saved data necessary for execution of the programs.The nonvolatile RAM 15 is a magnetic storage unit, a flash memory, anMRAM, FRAM (registered trademark), or an EEPROM, for example.Alternatively, the nonvolatile RAM 15 is an SRAM or a DRAM backed up bya battery, for example. The nonvolatile RAM 15 is configured as anonvolatile memory to hold its storage state even after the visualsensor controller 1 is powered off. The nonvolatile RAM 15 storessetting necessary for execution of the programs, for example. The framememory 12 stores image data.

[Model Pattern]

The nonvolatile RAM 15 includes a reference information storage 151 anda detection result storage 152. The reference information storage 151stores reference information (also called a “model pattern” or a“template”) indicating a subject (target mark 5). The referenceinformation may be a group of edge points (also called an “edge pointgroup”) in a subject (target mark 5), for example. The edge point is apoint where brightness changes largely in an image. For example, theedge point group may be formed by converting the subject (target mark 5)to an edge image through publicly-known Sobel filtering, and extractinga pixel (edge point) having an intensity of a predetermined threshold ormore from the edge image. The edge point group extracted in this wayfrom the image including the subject (target mark 5) to be detected isstored as the model pattern into the reference information storage 151.The model pattern is not limited to edge points. For example, featurepoints such as those extracted by publicly-known SIFT may be used as themodel pattern. Alternatively, the model pattern may be generated byarranging a geometric graphic such as a line segment, a rectangle, or acircle so as to match the contour of the subject (target mark 5). Inthis case, feature points may be provided at proper intervals on thegeometric graphic forming the contour. The model pattern may also be atemplate image generated by cutting out a part corresponding to a modelpattern designation area from a captured image of the target mark 5.Storing the model pattern in advance generated in the above-describedway into the reference information storage 151 is also called “teachingthe model pattern.” Teaching of the model pattern will be describedlater.

The detection result storage 152 stores a result of detection of thetarget mark 5 detected by using the taught model pattern from data aboutan image captured by the camera 2 while the target mark 5 having beenmoved in the set calibration range is at each destination.

In response to a command from the CPU 10, the camera 2 captures an imageof the subject to acquire the image, and outputs a signal about theacquired image. The camera interface 16 has the function of generating asynchronization signal for controlling timing of exposure for the camera2 in response to a command from the CPU 10, and the function ofamplifying a signal received from the camera 2. The camera 2 and thecamera interface 16 are not limited to any particular parts but arecommercially-available and common-used parts.

The signal about the image taken from the camera 2 is A/D converted bythe camera interface 16, and then stored temporarily as digital imagedata through the bus 11 into the frame memory 12. In the visual sensorcontroller 1, the CPU 10 processes the image by using data stored in theframe memory 12, the ROM 13, the RAM 14, and the nonvolatile RAM 15.Data resulting from the image processing is stored again into the framememory 12. In response to a command, the CPU 10 may transfer the datastored in the frame memory 12 to the monitor interface 17 and displaythe data on the monitor 19 in order to allow check of the substance ofthe data by an operator, for example.

The external equipment interface 18 is connected to various types ofexternal equipment. For example, the external equipment interface 18 isconnected to the robot controller 3 to receive a signal for triggeringimage processing from the robot controller 3 and supply the robotcontroller 3 with position information data obtained by imageprocessing, etc. A keyboard or a mouse may also be connected to theexternal equipment interface 18 as an input unit 33 for an operator, forexample.

The function of the CPU 10 will be described next in terms of eachprocessing unit. Description in terms of each processing step (method)will be omitted as each processing step (method) can be understood byreplacing “unit” in the following description by “step.” FIG. 4 is ablock diagram showing the functional configuration of the CPU 10 in thevisual sensor controller 1. The CPU 10 includes a model patterngeneration unit 101, a first parameter setting unit 102, a firstdetection unit 103, a first image range setting unit 105, a firstcalibration range measurement unit 106, an image capture control unit107, and a calibration unit 108. Each of these functional units andfunctional steps is realized by execution of a system program in the ROM13 by the CPU 10.

[Model Pattern Generation Unit 101]

The model pattern generation unit 101 generates a model pattern by usingthe camera 2, for example. FIG. 5 shows a flowchart followed by themodel pattern generation unit 101 for generating the model pattern. Thefunction of the model pattern generation unit 101 will be described byreferring to FIG. 5.

In step S1, the model pattern generation unit 101 exerts control to makethe camera 2 capture an image of the target mark 5 arranged in the viewof the camera 2. At this time, a relationship between the position ofthe camera 2 and that of the target mark 5 is desirably set to be thesame as that during detection of the target mark 5.

In step S2, the model pattern generation unit 101 sets an area in thecaptured image of the target mark 5 in the form of a rectangular area ora circular area, for example, as a model pattern designation area inwhich the target mark 5 appears. Further, the model pattern generationunit 101 defines a model pattern coordinate system (image coordinatesystem) in the model pattern designation area. FIG. 6 shows an exampleof the model pattern designation area, that of the model patterncoordinate system (image coordinate system), and that of the modelpattern. The model pattern generation unit 101 may set an areainstructed by an operator as the model pattern designation area.Alternatively, the model pattern generation unit 101 may determine spotsof large brightness gradients in the image as the contour of the imageof the target mark 5, and set the model pattern designation area so asto contain the contour of the image of the target mark 5 inside themodel pattern designation area.

In step S3, the model pattern generation unit 101 extracts an edge pointin the model pattern designation area as a feature point, obtainsphysical quantities such as the position of the edge point, thedirection and the magnitude of a brightness gradient at the edge point,etc., and converts the edge point to a value expressed in the modelpattern coordinate system defined in the model pattern designation area.Further, the model pattern generation unit 101 sets a point instructedby the operator as the point P where the three-dimensional positioninformation of the target mark 5 is to be measured, and stores the pointP into the reference information storage 151. By designating the point Pin advance where the three-dimensional position information of thetarget mark 5 is to be measured, the three-dimensional positioninformation and the three-dimensional posture of the target mark 5 aredetermined in advance in the end point coordinate system at the robot 4.If the point P where the three-dimensional position information of thetarget mark 5 is to be measured is not designated explicitly, the modelpattern generation unit 101 may set a center point of the model patternas a point where the three-dimensional position information of thetarget mark 5 is to be measured, for example. FIG. 7 shows an example ofthe point P where the three-dimensional position information of thetarget mark 5 is to be measured. In FIG. 7, a center point of the targetmark 5 is set as the point P where the three-dimensional positioninformation of the target mark 5 is to be measured. As described above,the model pattern is not limited to edge points. For example, featurepoints such as those extracted by publicly-known SIFT may be used as themodel pattern. Alternatively, the model pattern may be generated byarranging a geometric graphic such as a line segment, a rectangle, or acircle so as to match the contour of a subject (target mark 5) In thiscase, feature points may be provided at proper intervals on thegeometric graphic forming the contour. The model pattern may also be atemplate image generated by cutting out a part corresponding to themodel pattern designation area from the captured image of the targetmark 5.

In step S4, the model pattern generation unit 101 stores the generatedmodel pattern into the reference information storage 151. As describedabove, the model pattern is generated by using the image captured by thecamera 2.

The following describes a parameter for detecting the target mark 5 fromdata about the image captured by the camera 2 by using the model patternstored in the reference information storage 151.

A difference is generated in a distance between the camera 2 and theposition of the target mark 5 at a destination or in an angle from anoptical axis, for example. Hence, an image of the target mark 5 capturedby the camera 2 may differ in a manner that depends on a destination ofthe target mark 5 in the calibration range from the model patterngenerated by the model pattern generation unit 101 in terms ofappearance such as a size or the occurrence of rotation or distortion.As a result, it is likely that, in trying to detect the target mark 5completely matching the model pattern, it will be impossible to detectsuch a subject (target mark 5) from data about the image captured by thecamera 2.

[First Parameter Setting Unit 102]

The first parameter setting unit 102 sets a parameter for detecting thetarget mark 5 from data about an image captured by the camera 2 so as toallow detection of a model pattern at any destination of the target mark5 in the calibration range even in the above-described situation. Morespecifically, the parameter set by the first parameter setting unit 102depends on a detection algorithm. For example, the parameter may be setto have a predetermined range with respect to a model in terms of asize, rotation, distortion, a position range or an angle range fordetection, for example. Alternatively, the parameter may be set as asingle numerical value or an on/off value. The parameter is not limitedto these examples. Thus, the first detection unit 103 described later isconfigured to be capable of detecting the target mark 5 matching themodel pattern from captured image data by using a single numericalvalue, an on/off value, or a parameter value in a predetermined range soas to allow detection of the model pattern at any destination of thetarget mark 5. By doing so, the first detection unit 103 described laterbecomes capable of detecting the model pattern from the captured imagedata by using a proper parameter value. The parameter may be a parameterto be applied to data about an image captured by the camera 2. In thiscase, the first detection unit 103 is configured to detect the modelpattern about the target mark 5 from image data generated by applyingthe parameter to the data about the image captured by the camera 2. Forexample, the model pattern can be detected from image data generated bysmoothing the image data by a Gaussian filter. The parameter may be aparameter to be applied to the model pattern about the target mark 5. Inthis case, the first detection unit 103 is configured to detect thetarget mark 5 from the data about the image captured by the camera 2 byusing the model pattern to which the parameter is applied.

One example of the parameter set by the first parameter setting unit 102and applied to the model pattern may be a transformation matrix forprojection transformation, affine transformation, or homothetictransformation. For example, if a value of the parameter is set as asingle numerical value, a single transformation matrix is selectable. Ifa value of the parameter is set to have a predetermined range, atransformation matrix of a predetermined range is selectable. Thefollowing describes specific examples applied if the parameter is set tohave a predetermined range. In the case of a projection transformationmatrix, the parameter may be set in such a manner that a parameter rangecovers a projection transformation matrix with an element having adeviation of a predetermined threshold or less from a correspondingelement in a projection transformation matrix as a basis. In the case ofrotation, a range for a rotation angle may be set based on one rotationangle. Likewise, in the case of homothetic transformation, a range for ahomothetic ratio may be set based on one homothetic ratio. For example,based on an angle between the optical axis of the camera 2 and a planeon which the target mark 5 is to move during implementation ofcalibration (particularly if this plane is not at a right angle to theoptical axis but is tilted diagonally from the optical axis), a sizerange may be set from 0.9 to 1.1 for the camera 2 by using theparameter. By doing so, robust detection can be realized on theoccurrence of a difference in appearance of the target mark 5 resultingfrom different destinations of the target mark 5, for example. Aparameter value for exposure time is desirably set in consideration ofan angle between the camera 2 and a plane on which the target mark 5 isarranged or a relationship with illumination, for example.

[First Detection Unit 103]

The first detection unit 103 detects the target mark 5 from the dataabout the image captured by the camera 2, and measures the coordinateposition of the detected target mark 5 in an image coordinate system atthe camera 2. More specifically, the first detection unit 103 selects aparameter from a single numerical value, an on/off value, or apredetermined range for the parameter. In selecting the parameter fromthe predetermined range, a center value in the parameter range may beselected first, for example. Then, a value shifted in the plus or minusdirection from the center value may be selected as a next parameter, forexample.

If the parameter is a parameter to be applied to the data about theimage captured by the camera 2 as described above, after the firstdetection unit 103 selects the parameter from a single numerical value,an on/off value, or a predetermined range for the parameter, the firstdetection unit 103 converts the data about the image captured by thecamera 2 by using the selected parameter so as to allow detection of thetarget mark 5 from the image data. In this way, the first detection unit103 can detect the target mark 5 from the converted image data by thepublicly-known detection technique.

More specifically, the first detection unit 103 extracts a feature pointfrom the image data to which the parameter is applied by the same methodas applied for extracting a feature point from the taught model pattern,and conducts publicly-known matching between a feature point resultingfrom the conversion with the parameter and a feature point forming themodel pattern, thereby detecting the target mark 5.

Conversely, the first detection unit 103 may convert the model patternabout the target mark 5 by using the selected parameter. In this case,the first detection unit 103 can detect the target mark 5 matching theconverted model pattern from the captured image data by theabove-described publicly-known detection technique. More specifically,the first detection unit 103 extracts a feature point from the dataabout the image captured by the camera 2 by the same method as appliedfor extracting a feature point from the taught model pattern, andconducts publicly-known matching between the extracted feature point anda feature point in the model pattern to which the parameter is applied,thereby detecting the target mark 5. The first detection unit 103measures the coordinate position of the detected target mark 5 in theimage coordinate system at the camera 2.

[First Image Range Setting Unit 105]

As described above, in this embodiment, the target mark 5 is moved on aplane designated in advance. This plane may include multiple parallelplanes or may be formed by coupling multiple partial planes. Todetermine an operation range for the robot 4 (calibration range), thefirst image range setting unit 105 sets an image range in the imagecoordinate system at the camera 2 corresponding to this calibrationrange. More specifically, in response to an instruction from anoperator, the first image range setting unit 105 sets an image range inthe image captured by the camera 2 based on the image coordinate systemat the camera 2. In this case, the first image range setting unit 105can designate the image range as a rectangular range, for example. Thefirst image range setting unit 105 may set the image entirely as theimage range. If space for move of the target mark 5 includes anobstacle, for example, the first image range setting unit 105 may limitthe image range in response to an instruction from the operator so as toavoid the obstacle existing in the view of the camera 2. In this case,the first image range setting unit 105 may designate the image range asa closed graphic drawn with multiple line segments.

[First Calibration Range Measurement Unit 106]

The first calibration range measurement unit 106 measures a calibrationrange (a coordinate position in the robot coordinate system showing aboundary of the calibration range, for example) as an operation rangefor the robot 4 corresponding to the image range set by the first imagerange setting unit 105 before implementation of calibration bycontrolling the robot 4 so as to move the target mark 5 and making thefirst detection unit 103 detect the target mark 5. More specifically,the first calibration range measurement unit 106 determines whether ornot the target mark 5 is detectable in the image range designated in theimage coordinate system at the camera 2 by the first image range settingunit 105 by moving the target mark 5 and detecting the target mark 5repeatedly, thereby measuring a calibration range (a coordinate positionin the robot coordinate system showing a boundary of the calibrationrange, for example) as an operation range for the robot 4 (specifically,target mark 5) corresponding to the designated image range.

FIGS. 8A and 8B show flowcharts showing a flow followed by the firstcalibration range measurement unit 106 for measuring a coordinateposition in the robot coordinate system showing a boundary of acalibration range. The function of the first calibration rangemeasurement unit 106 will be described by referring to FIGS. 8A and 8B.A plane on which the target mark 5 is to be moved during implementationof calibration is defined as an XY plane in a three-dimensionalcoordinate system designated in advance. However, this plane is notlimited to the XY plane in the three-dimensional coordinate system.

In step S11, the first calibration range measurement unit 106 acquiresthe three-dimensional coordinate position of the target mark 5 in therobot coordinate system arranged at an initial position on a plane inthe view of the camera 2 and the coordinate position of this target mark5 in the image coordinate system at the camera 2 through the robotcontroller 3 and the first detection unit 103 respectively. The targetmark 5 may be moved to the initial position in response to jog operationby an operator, for example. Alternatively, the target mark 5 may bemoved to an initial position stored in the robot controller 3.

In step S12, the first calibration range measurement unit 106 moves therobot 4 (specifically, target mark 5) through the robot controller 3 onthe plane from the initial position by a unit stroke in an X directionset in advance in an X-axis direction. Then, the first calibration rangemeasurement unit 106 acquires the three-dimensional coordinate positionof the target mark 5 in the robot coordinate system and the coordinateposition of the target mark 5 in the image coordinate system at thecamera 2 through the robot controller 3 and the first detection unit 103respectively.

In step S13, the first calibration range measurement unit 106 moves therobot 4 (specifically, target mark 5) through the robot controller 3 onthe plane from the initial position by a unit stroke in a Y directionset in advance in a Y-axis direction. Then, the first calibration rangemeasurement unit 106 acquires the three-dimensional coordinate positionof the target mark 5 in the robot coordinate system and the coordinateposition of the target mark 5 in the image coordinate system at thecamera 2 through the robot controller 3 and the first detection unit 103respectively.

In step S14, the first calibration range measurement unit 106 calculatesa three-dimensional vector VX (magnitude and direction) in the robotcoordinate system corresponding to the X-direction unit stroke in theX-axis direction on the plane, and a two-dimensional vector vx(magnitude and direction) in the image coordinate system at the camera 2showing the stroke of the target mark 5. The first calibration rangemeasurement unit 106 further calculates a three-dimensional vector VY(magnitude and direction) in the robot coordinate system correspondingto the Y-direction unit stroke in the Y-axis direction on the plane, anda two-dimensional vector vy (magnitude and direction) in the imagecoordinate system at the camera 2 showing the stroke of the target mark5. By doing so, the first calibration range measurement unit 106 canacquire approximate association between the stroke in the imagecoordinate system at the camera 2 and the stroke in the robot coordinatesystem. For example, the first calibration range measurement unit 106can convert a position information in the image coordinate system to anapproximate position information in the robot coordinate system by usinga transformation matrix R calculated based on the following formula (1):

$\begin{matrix}{R = {\left\lbrack {{VX}\mspace{14mu}{VY}\mspace{14mu} 0} \right\rbrack\begin{bmatrix}{vx} & {vy} & 0 \\0 & 0 & 1\end{bmatrix}}^{- 1}} & \left\lbrack {{Formula}\mspace{14mu} 1} \right\rbrack\end{matrix}$Regarding move by the Y-direction unit stroke set in advance in theY-axis direction in step S13, a start point for the vector is notlimited to the initial position. The vector may be started from a pointto which the robot 4 has been moved from the initial position by theX-direction unit stroke in step S12.

In step S15, based on the association calculated in step S14, the firstcalibration range measurement unit 106 calculates and sets the magnitudeand the direction (V) of the stroke in the robot coordinate system fromthe initial position on the plane corresponding to the magnitude and thedirection (v) of the stroke in the image coordinate system at the camera2 from the initial position to a boundary position (corner, for example)in the image coordinate system at the camera 2. At this time, themagnitude and the direction (V) are desirably set so as to increase theprobability of detection of the target mark 5 in a range in the imagecoordinate system at the camera 2 by being multiplied by a factor lessthan 1. Coordinate values in the image range in the image coordinatesystem at the camera 2 (boundary positions including four corners, forexample) are calculated in advance.

In step S16, the first calibration range measurement unit 106 moves thetarget mark 5. More specifically, based on the set magnitude and the setdirection (V) of the stroke, the first calibration range measurementunit 106 moves the robot 4 (specifically, target mark 5) through therobot controller 3 from the initial position on the plane.

In step S17, the first calibration range measurement unit 106 determinesthrough the first detection unit 103 whether or not the target mark 5 isdetectable. If the target mark 5 has been detected successfully (Yes),the flow goes to step S18. If the target mark 5 has not been detectedsuccessfully (No), the flow goes to step S19.

In step S18, the first calibration range measurement unit 106 determinesthrough the first detection unit 103 whether or not a distance from thecoordinate position of the target mark 5 in the image coordinate systemat the camera 2 to the boundary position (corner, for example) in theimage coordinate system at the camera 2 is small (specifically, whetheror not the distance is a predetermined threshold or less). If thedistance is determined to be small (Yes), the flow goes to step S20. Ifthe distance is determined not to be small (No), the flow goes to stepS22.

In step S19, the first calibration range measurement unit 106 sets themagnitude and the direction of the stroke again. More specifically, ifthe first calibration range measurement unit 106 determines that thetarget mark 5 has not been detected successfully during move from theinitial position, the first calibration range measurement unit 106multiplies the set magnitude and the set direction (V) of the stroke bya factor less than 1, thereby setting the magnitude and the direction(V) of the stroke again. The factor used in this step should be lowerthan the factor having been applied to the move from the initialposition. Then, the flow goes to step S16.

In step S20, the first calibration range measurement unit 106 stores acurrent position information of the target mark 5 in the robotcoordinate system and a current position information of the target mark5 in the image coordinate system at the camera 2.

In step S21, the first calibration range measurement unit 106 determineswhether or not measurement has been made for all the boundary positions.If measurement has been made for all the boundary positions (Yes), theflow ends. If there is a boundary position to be subject to nextmeasurement (if No), the flow goes to step S15.

In step S22, the first calibration range measurement unit 106 sets themagnitude and the direction (V′) of the stroke from the current positionon the plane in the robot coordinate system corresponding to themagnitude and the direction (v′) of the stroke from the current positionto the boundary position.

In step S23, based on the set magnitude and the set direction (V′) ofthe stroke from the current position, the first calibration rangemeasurement unit 106 moves the robot 4 (specifically, target mark 5).

In step S24, the first calibration range measurement unit 106 determinesthrough the first detection unit 103 whether or not the target mark 5 isdetectable. If the target mark 5 has been detected successfully (Yes),the flow goes to step S18. If the target mark 5 has not been detectedsuccessfully (No), the flow goes to step S25.

In step S25, the first calibration range measurement unit 106 multipliesthe set magnitude and the set direction (V′) of the stroke by a factorless than 1, thereby setting the magnitude and the direction (V′) of thestroke again from a position where the robot 4 existed before moving.The factor used in this step should be lower than the factor having beenapplied to the move from the position where the robot 4 existed beforemoving. Then, the flow goes to step S23. As described above, the firstcalibration range measurement unit 106 can measure a coordinate positionin the robot coordinate system showing a boundary of a calibration rangein which the target mark 5 is to be detected. The above-describedflowchart is not restrictive but has been described as an example.During implementation of calibration, the visual sensor controller 1functioning as the calibration device controls the robot 4 so as to movethe target mark 5 in the calibration range. At this time, the arm 41desirably moves through the set calibration range uniformly. Forexample, the visual sensor controller 1 may control the robot 4 in sucha manner that the target mark 5 moves by tracing a path shown in FIG. 9Aor 9B.

[Image Capture Control Unit 107]

The image capture control unit 107 makes the camera 2 capture an imageof the target mark 5 attached to the end of the arm 41 of the robot 4and moved by the robot 4 in a calibration range at each of multipledestination positions during implementation of calibration. The numberof the destinations is desirably set to be larger than a number minimumfor allowing calibration (a desired number is 20 or more, for example).By doing so, calibration can be performed more precisely.

[Calibration Unit 108]

The calibration unit 108 stores the following coordinate positions intothe detection result storage 152 at each of multiple destinations of thetarget mark 5 attached to the end of the arm 41 of the robot 4 and movedby the robot controller 3 during implementation of calibration: thecoordinate position of the target mark 5 in the image coordinate systemat the camera 2 appearing in data about an image captured by the camera2; and the coordinate position of the target mark 5 in the robotcoordinate system while the image of the target mark 5 is captured bythe camera 2. Then, the calibration unit 108 calibrates the camera 2based on the coordinate position of the target mark 5 in the imagecoordinate system at the camera 2 stored in the detection result storage152, and the coordinate position of the target mark 5 in the robotcoordinate system while the image of the target mark 5 is captured bythe camera 2 stored in the detection result storage 152.

FIG. 10 is a flowchart showing calibration process on the camera 2executed by the visual sensor controller 1 (CPU 10) according to thisembodiment.

In step S31, a model pattern is generated and a parameter is set.

In step S32, the CPU 10 (first image range setting unit 105) sets animage range (corresponding to a calibration range) for the camera 2 inthe image coordinate system in response to designation by an operator.The operator can designate the image range (corresponding to thecalibration range) in an image captured by the camera 2.

In step S33, the first calibration range measurement unit 106 measures acoordinate position in the robot coordinate system showing a boundary ofthe calibration range. More specifically, as described above, the firstcalibration range measurement unit 106 determines whether or not thetarget mark 5 is detectable in the image range by moving the target mark5 and detecting the target mark 5 repeatedly, thereby measuring thecalibration range (a coordinate position in the robot coordinate systemshowing a boundary of an operation range for the robot 4, for example).(The process in step S33 is performed based on the above-describedprocess flow followed by the first calibration range measurement unit106.)

In step S34, the CPU 10 (calibration unit 108) sets a measurementcounter for counting the number of times measurement is made at 1.

In step S35, the CPU 10 (calibration unit 108) acquires thethree-dimensional coordinate position of the target mark 5 in the robotcoordinate system measured by the robot controller 3.

In step S36, the CPU 10 (first detection unit 103) detects the targetmark 5 from data about the image captured by the camera 2, and measuresthe coordinate position of the detected target mark 5 in the imagecoordinate system at the camera 2.

In step S37, the CPU 10 (calibration unit 108) stores thethree-dimensional coordinate position of the target mark 5 in the robotcoordinate system and the coordinate position of the target mark 5 inthe image coordinate system at the camera 2 in association with eachother while the target mark 5 is at a current position.

In step S38, the CPU 10 (calibration unit 108) increments themeasurement counter for counting the number of times measurement is madeby 1.

In step S39, it is determined whether or not the measurement counterexceeds a predetermined value. If the measurement counter exceeds thepredetermined value (Yes), the flow goes to step S41. If the measurementcounter does not exceed the predetermined value (No), the flow goes tostep S40.

In step S40, the robot controller 3 moves the target mark 5 attached tothe end of the arm 41 of the robot 4 to a place in the calibration rangewhere the target mark 5 can be measured from the camera 2. Then, theflow goes to step S35.

In step S41, the CPU 10 (calibration unit 108) calibrates the camera 2based on the association stored in step S37 between thethree-dimensional coordinate position of the target mark 5 in the robotcoordinate system and the coordinate position of the target mark 5 inthe image coordinate system at the camera 2.

This process flow is not restrictive but has been described as anexample.

As described above, the visual sensor controller 1 of the firstembodiment includes: the first image range setting unit 105 that sets animage range in the image coordinate system at the camera 2; and thefirst calibration range measurement unit 106 that measures a calibrationrange as an operation range for the robot 4 corresponding to the setimage range before implementation of calibration by controlling therobot 4 to move the target mark 5 and detecting the target mark 5. Inthis way, an operation range for the robot 4 (calibration range)corresponding to a range of move of the target mark 5 is measured inadvance before implementation of calibration based on a view range for asingle visual sensor, so that the calibration can be performed with anincreased degree of freedom. Further, the target mark 5 is allowed tomove in a range covering the view of the camera 2 entirely, so that thecalibration can be performed more precisely. If a plane on which thetarget mark 5 is to move includes an obstacle, for example, an imagerange is designated in response to an instruction from an operator so asto avoid the obstacle existing in the view of the camera 2. By doing so,the calibration range can be set efficiently.

A calibration range can be set on a plane. The plane on which the targetmark 5 is to be moved during implementation of calibration can be aplane vertical to the optical axis of the camera 2 or a plane tiltedfrom the optical axis of the camera 2 as shown in FIG. 11A. Further, asshown in FIG. 11B, a calibration range can be set on each of at leasttwo planes. A calibration range can be extended by making such settings.This increases the degree of freedom further in calibration, so that thecalibration is performed more precisely.

A calibration method of the first embodiment and a program of the firstembodiment achieve effects comparable to those achieved by the visualsensor controller 1 of the first embodiment.

The first embodiment of the present invention has been described, butthe present invention is not limited to the above-described firstembodiment. The effects described in the first embodiment are merely alist of most preferred effects resulting from the present invention.Effects achieved by the present invention are not limited to thosedescribed in the first embodiment.

[First Modification]

In the first embodiment, the model pattern generation unit 101 isconfigured to generate a model pattern by using the camera 2 and storethe generated model pattern. Instead of being configured to generate amodel pattern by using the camera 2, the model pattern generation unit101 may be configured to use an existing shape (a shape such as acircle, for example) as the target mark 5.

[Second Modification]

In the first embodiment, the target mark 5 is moved on a planedesignated in advance and a calibration range is measured on this plane.Alternatively, the calibration range may be defined in anythree-dimensional space. This is expected to increase the degree offreedom further in calibration. For example, the plane to be designatedin advance may include multiple parallel planes or may be formed bycoupling multiple partial planes. As shown in FIG. 11B, for example, theplane to be designated may include at least two planes. If the planeincludes multiple planes, the first calibration range measurement unit106 measures a coordinate position in the robot coordinate systemshowing a boundary of a calibration range on each of the planes. Asanother example, two planes may be defined in a three-dimensionalcoordinate system. A calibration range may be set as space inside ahexahedron defined by a point of intersection between each plane and aline of sight pointing to a corner in an image captured by the camera 2.In this case, the first calibration range measurement unit 106 measuresa coordinate position in the robot coordinate system showing a boundaryof a calibration range in this hexahedron.

Second Embodiment

A second embodiment will be described next. In the second embodiment, avisual sensor controller 1A functioning as a calibration device sets acalibration range as an operation range for a robot 4 for allowing imagecapture of a target mark 5 in an image range for a stereo camera 2Aincluding at least a first camera 21 and a second camera 22 after thetarget mark 5 is moved. The description of the second embodiment doesnot cover structures and functions common to those of the firstembodiment but is directed to issues peculiar to the second embodiment.

FIG. 12 shows the configuration of a robot system 1000 entirely forcalibration on a stereo camera. The robot system 1000 is for performingcalibration using the stereo camera 2A with the first camera 21 and thesecond camera 22. The number of cameras forming the stereo camera 2A isnot limited to two but can be any number of two or more. Each of thecameras forming the stereo camera 2A is certainly applicable as a singlecamera. The stereo camera 2A is fixed to a pedestal (not shown in thedrawings). As shown in FIG. 13A, the first camera 21 and the secondcamera 22 may be arranged parallel to each other. As shown in FIG. 13B,each of the first camera 21 and the second camera 22 may be arranged ata tilt. Tilting each of the first camera 21 and the second camera 22makes it possible to increase an area of an overlap between an area ofimage captured by the first camera 21 and an area of image captured bythe second camera 22, compared to arranging the first camera 21 and thesecond camera 22 parallel to each other. Specifically, tilting each ofthe first camera 21 and the second camera 22 makes it possible toincrease an area where three-dimensional measurement is allowed by thestereo camera 2A, compared to arranging the first camera 21 and thesecond camera 22 parallel to each other. The first camera 21 and thesecond camera 22 desirably have the same configuration in terms of aview range, a lens, etc. By doing so, the first camera 21 and the secondcamera 22 are expected to catch the target mark 5 in the same way.

[Visual Sensor Controller 1A]

The stereo camera 2A is connected to the visual sensor controller 1A.The visual sensor controller 1A makes the first camera 21 and the secondcamera 22 capture images of the target mark 5, and calibrates each ofthe first camera 21 and the second camera 22.

The functional configuration of the visual sensor controller 1Afunctioning as the calibration device is the same as the functionalconfiguration of the visual sensor controller 1 of the first embodiment(FIG. 3), except that the stereo camera 2A (first camera 21 and secondcamera 22) is connected to the bus 11 through the camera interface 16.The functional configuration of the visual sensor controller 1A can beunderstood by referring to FIG. 3 mentioned above.

FIG. 14 is a block diagram showing the functional configuration of a CPU10 in the visual sensor controller 1A. As shown in FIG. 14, the CPU 10in the visual sensor controller 1A includes a model pattern generationunit 101, a first parameter setting unit 102, a first detection unit103, a first image range setting unit 105, a first calibration rangemeasurement unit 106, a second parameter setting unit 1022, a seconddetection unit 1032, a second image range setting unit 1052, a secondcalibration range measurement unit 1062, an image capture control unit107, and a calibration unit 108.

The model pattern generation unit 101 generates a model pattern bymaking the first camera 21 capture an image of the target mark 5arranged in the view of the first camera 21. The model pattern generatedby using the image captured by the first camera 21 is also used as amodel pattern for the second camera 22. A model pattern for the secondcamera 22 may be generated individually by making the second camera 22capture an image of the target mark 5 arranged in the view of the secondcamera 22.

The first parameter setting unit 102 sets a first parameter fordetecting a model pattern about the target mark 5 from data about animage captured by the first camera 21. The function of the firstparameter setting unit 102 is comparable to that of the first parametersetting unit 102 in the first embodiment.

[Second Parameter Setting Unit 1022]

The following description is for the second parameter setting unit 1022that uses the model pattern about the target mark 5 generated by usingthe first camera 21 to set a second parameter for detecting this modelpattern from data about an image captured by the second camera 22. Thesecond parameter setting unit 1022 sets the second parameter fordetecting the model pattern about the target mark 5 from the data aboutthe image captured by the second camera 22 based on the first parameter.More specifically, the second parameter setting unit 1022 uses the firstparameter as it is for initially setting a value of the secondparameter. Meanwhile, if a value of the second parameter is set to havea predetermined range during initial setting of the second parameter,for example, the second parameter setting unit 1022 may employ the samerange as the first parameter. Alternatively, the second parametersetting unit 1022 may set a wide range for the second parameter coveringa predetermined range for the first parameter set by the first parametersetting unit 102. In such cases, if the second detection unit 1032described later has detected the model pattern about the target mark 5successfully from the data about the image captured by the second camera22 by applying a given value of the second parameter, the secondparameter setting unit 1022 can set a range for a value of the secondparameter again with respect to this value of the second parameter as acenter based on a deviation from a center value in the predeterminedrange for the first parameter. For example, if the first parameter has asize range from 0.9 to 1.1, a center value is 1.0 and a deviation fromthe center value in the predetermined range for the first parameter is0.1. If a subject (target mark 5) of the second camera 22 has beendetected successfully by setting a size of the second parameter for thesecond camera 22 at 0.95, a center value of the second parameter is setat 0.95 and the deviation in the first parameter is applied to thesecond parameter. Specifically, a value of the second parameter is setin a range [0.85 to 1.05] with respect to 0.95 as a center value. Inthis way, the range for a value of the second parameter during initialsetting of the second parameter can be readjusted, so that the modelpattern can be detected more efficiently from the data about the imagecaptured by the second camera 22. The second parameter is not limited tothese examples.

[First Detection Unit 103]

The first detection unit 103 detects the model pattern about the targetmark 5 from the data about the image captured by the first camera 21,and measures the coordinate position of the detected target mark 5 in animage coordinate system at the first camera 21. The function of thefirst detection unit 103 is comparable to that of the first detectionunit 103 in the first embodiment.

[Second Detection Unit 1032]

The second detection unit 1032 detects the model pattern about thetarget mark 5 from the data about the image captured by the secondcamera 22, and measures the coordinate position of the detected targetmark 5 in an image coordinate system at the second camera 22. Detectionprocess by the second detection unit 1032 can be understood by replacingthe camera 2, the first parameter setting unit 102, and the parameter inthe description given above in the first embodiment relating to thefirst detection unit 103 by the second camera 22, the second parametersetting unit 1022, and the second parameter respectively.

[First Image Range Setting Unit 105]

In response to an instruction from an operator, the first image rangesetting unit 105 sets a first image range in the image captured by thefirst camera 21 based on the image coordinate system at the first camera21. The first image range may cover the captured image entirely. Thefirst image range setting unit 105 can designate the first image rangeas a rectangular range, for example. If a plane on which the target mark5 attached to an end portion of an arm 41 is to move includes anobstacle, for example, the first image range setting unit 105 may limitthe first image range in response to an instruction from the operator soas to avoid the obstacle existing in the view of the first camera 21. Inthis case, the first image range setting unit 105 may designate thefirst image range as a closed graphic drawn with multiple line segments.The function of the first image range setting unit 105 is comparable tothat of the first image range setting unit 105 in the first embodiment.

[Second Image Range Setting Unit 1052]

Like the first image range setting unit 105, the second image rangesetting unit 1052 sets a second image range in the image captured by thesecond camera 22 based on the image coordinate system at the secondcamera 22. Process by the second image range setting unit 1052 can beunderstood by replacing the camera 2 in the description given above inthe first embodiment relating to the first image range setting unit 105by the second camera 22.

[First Calibration Range Measurement Unit 106]

The first calibration range measurement unit 106 measures a firstcalibration range as an operation range for the robot 4 corresponding tothe first image range set by the first image range setting unit 105before implementation of calibration by controlling the robot 4 so as tomove the target mark 5 and making the first detection unit 103 detectthe target mark 5. More specifically, the first calibration rangemeasurement unit 106 determines whether or not the target mark 5 isdetectable in the first image range by the first camera 21 by moving thetarget mark 5 and detecting the target mark 5 repeatedly, therebymeasuring the first calibration range as an operation range for therobot 4 corresponding to the first image range. The function of thefirst calibration range measurement unit 106 is comparable to that ofthe first calibration range measurement unit 106 in the firstembodiment.

[Second Calibration Range Measurement Unit 1062]

Likewise, the second calibration range measurement unit 1062 determineswhether or not the target mark 5 is detectable in the second image rangeby the second camera 22 by moving the target mark 5 and detecting thetarget mark 5 repeatedly, thereby measuring a second calibration rangeas an operation range for the robot 4 corresponding to the second imagerange. Process by the second calibration range measurement unit 1062 canbe understood by replacing the camera 2, the first parameter settingunit 102, the parameter, and the image range in the description givenabove in the first embodiment relating to the first calibration rangemeasurement unit 106 by the second camera 22, the second parametersetting unit 1022, the second parameter, and the second image rangerespectively. Like in the first embodiment, the target mark 5 is movedon a plane designated in advance in this embodiment. The firstcalibration range and the second calibration range are set on the sameplane. A flowchart about process of measuring the first calibrationrange followed by the first calibration range measurement unit 106, anda flowchart about process of measuring the second calibration rangefollowed by the second calibration range measurement unit 1062, can beunderstood by the replacing the camera 2, the first parameter settingunit 102, the parameter, and the image range referred to in thedescription in the first embodiment for the flowchart relating to thecalibration range measurement unit 106 (FIGS. 8A and 8B) by the firstcamera 21, the first parameter setting unit 102, the first parameter,and the first image range respectively, and by the second camera 22, thesecond parameter setting unit 1022, the second parameter, and the secondimage range respectively. While the first camera 21 and the secondcamera 22 are calibrated, the visual sensor controller 1A functioning asthe calibration device controls the robot 4 so as to move the targetmark 5 in at least one of the first calibration range and the secondcalibration range, or in a range covered by both the first calibrationrange and the second calibration range. Specifically, the target mark 5may be moved only in a range covered by both the first calibration rangeand the second calibration range, only in a range covered by either thefirst calibration range or the second calibration range, or in a rangeincluding the first calibration range and the second calibration range.

[Image Capture Control Unit 107]

The image capture control unit 107 makes the first camera 21 and thesecond camera 22 capture images of the target mark 5 at each of multipledestination positions attached to the end of the arm 41 of the robot 4and to be moved by the robot 4 in at least one of the first calibrationrange and the second calibration range, or in a range covered by boththe first calibration range and the second calibration range duringimplementation of calibration. The number of the destinations isdesirably set to be larger than a number minimum for allowingcalibration (a desired number is 20 or more, for example) on each of thefirst camera 21 and the second camera 22. By doing so, calibration canbe performed more precisely.

[Calibration Unit 108]

The calibration unit 108 stores the following coordinate positions intothe detection result storage 152 at each of multiple destinations of thetarget mark 5 moved in at least one of the first calibration range andthe second calibration range, or in a range covered by both the firstcalibration range and the second calibration range during implementationof calibration: the coordinate position of the target mark 5 in theimage coordinate system at the first camera 21; the coordinate positionof the target mark 5 in the image coordinate system at the second camera22; and the coordinate position of the target mark 5 in the robotcoordinate system while the images of the target mark 5 are captured bythe first camera 21 and the second camera 22. Then, the calibration unit108 calibrates the first camera 21 and the second camera 22 based on thecoordinate position of the target mark 5 in the image coordinate systemat the first camera 21 stored in the detection result storage 152, thecoordinate position of the target mark 5 in the image coordinate systemat the second camera 22 stored in the detection result storage 152, andthe coordinate position of the target mark 5 in the robot coordinatesystem while the images of the target mark 5 are captured by the firstcamera 21 and the second camera 22 stored in the detection resultstorage 152. The first camera 21 and the second camera 22 may becalibrated individually.

FIG. 15 is an example of a flowchart showing calibration process on thefirst camera 21 and the second camera 22 performed by the visual sensorcontroller 1A (CPU 10) according to this embodiment. This flowchartshows process performed if the target mark 5 is to be moved in a rangeincluding the first calibration range and the second calibration range.

In step S51, a model pattern is generated. Further, the first parameterand the second parameter are set.

In step S52, in response to designation by an operator, the CPU 10(first image range setting unit 105 and second image range setting unit1052) sets the first image range in an image captured by the firstcamera 21 in the image coordinate system at the first camera 21, andsets the second image range in an image captured by the second camera 22in the image coordinate system at the second camera 22.

In step S53, the CPU 10 (first calibration range measurement unit 106and second calibration range measurement unit 1062) measures acoordinate position in the robot coordinate system showing a boundary ofthe first calibration range corresponding to the first image range forthe first camera 21, and measures a coordinate position in the robotcoordinate system showing a boundary of the second calibration rangecorresponding to the second image range for the second camera 22. Inthis way, the first calibration range and the second calibration rangeare set to allow calibration process on the first camera 21 and thesecond camera 22 to be started.

[Calibration Process]

In step S54, the CPU 10 (calibration unit 108) sets a measurementcounter for counting the number of times measurement is made at 1.

In step S55, the CPU 10 (calibration unit 108) acquires thethree-dimensional coordinate position of the target mark 5 in the robotcoordinate system measured by the robot controller 3.

In step S56, the CPU 10 (first detection unit 103 and second detectionunit 1032) detects the target mark 5 from data about the image capturedby the first camera 21 and from data about the image captured by thesecond camera 22, and measures the coordinate position of the detectedtarget mark 5 in each of the image coordinate system at the first camera21 and the image coordinate system at the second camera 22. The targetmark 5 may be detected only from the first camera 21 and the secondcamera 22. In this case, only a coordinate position in an imagecoordinate system at a camera having detected the target mark 5successfully is stored.

In step S57, the CPU 10 (calibration unit 108) stores thethree-dimensional coordinate position of the target mark 5 in the robotcoordinate system, the coordinate position of the target mark 5 in theimage coordinate system at the first camera 21, and the coordinateposition of the target mark 5 in the image coordinate system at thesecond camera 22 in association with each other while the target mark 5is at a current position.

In step S58, the CPU 10 (calibration unit 108) increments themeasurement counter for counting the number of times measurement is madeby 1.

In step S59, if the measurement counter does not exceed a predeterminedvalue (No), the flow goes to step S60. If the measurement counterexceeds the predetermined value (Yes), the flow goes to step S61.

In step S60, the robot controller 3 moves the target mark 5 attached tothe end of the arm 41 of the robot 4 to a place set in advance in thefirst calibration range or the second calibration range where the targetmark 5 can be measured from at least one of the first camera 21 and thesecond camera 22. Then, the flow goes to step S55.

In step S61, the CPU 10 (calibration unit 108) calibrates the firstcamera 21 and the second camera 22 based on the association stored instep S57 between the three-dimensional coordinate position of the targetmark 5 in the robot coordinate system, the coordinate position of thetarget mark 5 in the image coordinate system at the first camera 21, andthe coordinate position of the target mark 5 in the image coordinatesystem at the second camera 22. This process flow is not restrictive buthas been described as an example. For example, if the target mark 5 isto be moved only in a range covered by both the first calibration rangeand the second calibration range, steps S56 and S60 can be replaced asfollows. In step S56, only if the target mark 5 is detected from boththe first camera 21 and the second camera 22, the coordinate position ofthe target mark 5 in the image coordinate system at the first camera 21and the coordinate position of the target mark 5 in the image coordinatesystem at the second camera 22 may be stored. In step S60, the robotcontroller 3 moves the target mark 5 attached to the end of the arm 41of the robot 4 to a place set in advance in the first calibration rangeand the second calibration range where the target mark 5 can be measuredfrom both the first camera 21 and the second camera 22.

The visual sensor controller 1A of the second embodiment includes: thefirst image range setting unit 105 that sets the first image range inthe image coordinate system at the first camera 21; the firstcalibration range measurement unit 106 that measures the firstcalibration range as an operation range for the robot 4 corresponding tothe first image range before implementation of calibration bycontrolling the robot 4 to move the target mark 5 and detecting thetarget mark 5; the second image range setting unit 1052 that sets thesecond image range in the image coordinate system at the second camera22; and the second calibration range measurement unit 1062 that measuresthe second calibration range as an operation range for the robot 4corresponding to the second image range before implementation of thecalibration by controlling the robot 4 to move the target mark 5 anddetecting the target mark 5. In this way, an operation range for therobot 4 (first calibration range and second calibration range)corresponding to a range of move of the target mark 5 is measured inadvance before implementation of the calibration based on view rangesfor multiple cameras (visual sensors), so that the calibration can beperformed with an increased degree of freedom. Further, the target mark5 is allowed to move in a maximum range for each camera, so that thecalibration can be performed more precisely. If space in which thetarget mark 5 is to move includes an obstacle, for example, the firstimage range and the second image range are designated in response to aninstruction from an operator so as to avoid the obstacle. By doing so,the operation range for the robot 4 (first calibration and secondcalibration range) can be set efficiently.

Each of the first calibration range and the second calibration range canbe configured to be set on a plane. A calibration method of the secondembodiment and a program of the second embodiment achieve effectscomparable to those achieved by the visual sensor controller 1A of thesecond embodiment.

The present invention is not limited to the above-described secondembodiment. The effects described in the second embodiment are merely alist of most preferred effects resulting from the present invention.Effects achieved by the present invention are not limited to thosedescribed in the second embodiment.

[First Modification]

In the second embodiment, the stereo camera 2A includes two cameras.Alternatively, the stereo camera 2A may be configured to include threeor more cameras.

[Second Modification]

In the second embodiment, the model pattern generation unit 101 isconfigured to generate a model pattern by using the first camera 21 andstore the generated model pattern. Alternatively, the model patterngeneration unit 101 may be configured to generate a model pattern byusing the second camera 22 and store the generated model pattern. In thesecond embodiment, the model pattern generation unit 101 is configuredto generate and store a model pattern. Instead of being configured togenerate a model pattern by using the first camera 21 or the secondcamera 22, the model pattern generation unit 101 may be configured touse an existing shape (a shape such as a circle, for example) as thetarget mark 5.

[Third Modification]

In the second embodiment, the target mark 5 is moved on a planedesignated in advance. Further, the first calibration range and thesecond calibration range are measured on this plane. Alternatively, thefirst calibration range and the second calibration range may be definedin any three-dimensional space. For example, the plane to be designatedin advance may include multiple parallel planes or may be formed bycoupling multiple partial planes.

In these embodiments, the visual sensor controller 1 or 1A functions asthe calibration device. However, the calibration device is not limitedto the visual sensor controller 1 or 1A. The calibration device may be acontroller including the visual sensor controller 1 or 1A and the robotcontroller 3 integrated with each other. Alternatively, the calibrationdevice may cover information processing devices (computers) in general.For example, the calibration device may be a server, a PC, various typesof controllers, etc.

The calibration method implemented by the visual sensor controller 1 or1A is realized by software. To realize the calibration method bysoftware, programs constituting the software are installed on a computer(visual sensor controller 1). These programs may be stored in aremovable medium and then distributed to a user. Alternatively, theseprograms may be distributed by being downloaded onto a computer of theuser through a network.

EXPLANATION OF REFERENCE NUMERALS

-   1000 Robot system-   1 Visual sensor controller (calibration device)-   1A Visual sensor controller (calibration device)-   10 CPU-   101 Model pattern generation unit-   102 First parameter setting unit-   1022 Second parameter setting unit-   103 First detection unit-   1032 Second detection unit-   105 First image range setting unit-   1052 Second image range setting unit-   106 First calibration range measurement unit-   1062 Second calibration range measurement unit-   107 Image capture control unit-   108 Calibration unit-   11 Bus-   12 Frame memory-   13 ROM-   14 RAM-   15 Nonvolatile memory-   151 Reference information storage-   152 Detection result storage-   16 Camera interface-   17 Monitor interface-   18 External equipment interface-   19 Monitor-   2 Camera-   2A Stereo camera-   21 First camera-   22 Second camera-   3 Robot controller-   4 Robot-   41 Arm-   5 Target mark

What is claimed is:
 1. A calibration device that associates a robotcoordinate system at a robot and an image coordinate system at a cameraby placing a target mark at the robot, controlling the robot so as tomove the target mark, and detecting the target mark at multiple pointsin a view of the camera, the calibration device comprising: a memoryconfigured to store a program; and a processor configured to execute theprogram and control the calibration device to: set an image range in theimage coordinate system; and measure a calibration range as an operationrange for the robot corresponding to the image range beforeimplementation of calibration by controlling the robot to move thetarget mark and detecting the target mark by determining whether or notthe target mark is detectable in the image range that was set in theimage coordinate system by moving the target mark and detecting thetarget mark repeatedly, thereby measuring a calibration range as anoperation range for the robot corresponding to the set image range,wherein during implementation of the calibration, the robot iscontrolled so as to move the target mark in the calibration range. 2.The calibration device according to claim 1, wherein the calibrationrange is set on a plane.
 3. The calibration device according to claim 1,wherein the calibration range is set on a plane that is not tilted froman optical axis of the camera.
 4. The calibration device according toclaim 1, wherein the calibration range is set on a plane tilted from anoptical axis of the camera.
 5. The calibration device according to claim1, wherein the calibration range is set on each of at least two planes.6. The calibration device according to claim 1, wherein the image rangein the image coordinate system is set so as to avoid a target markobstacle being in the view of the camera; and the calibration range ismeasured as the operation range for the robot corresponding to the setimage range so that a target mark obstacle is avoided in the measuredcalibration range.
 7. A calibration method executed by a calibrationdevice including a memory storing a program and a processor, thecalibration device associating a robot coordinate system at a robot andan image coordinate system at a camera by placing a target mark at therobot, controlling the robot so as to move the target mark, anddetecting the target mark at multiple points in a view of the camera,the processor executing the program to perform operations of thecalibration method comprising: setting an image range in the imagecoordinate system; and measuring a calibration range as an operationrange for the robot corresponding to the image range beforeimplementation of calibration by controlling the robot to move thetarget mark and detecting the target mark by determining whether or notthe target mark is detectable in the image range that was set in theimage coordinate system by moving the target mark and detecting thetarget mark repeatedly, thereby measuring a calibration range as anoperation range for the robot corresponding to the set image range,wherein during implementation of the calibration, the robot iscontrolled so as to move the target mark in the calibration range. 8.The calibration method according to claim 7, wherein the image range inthe image coordinate system is set so as to avoid a target mark obstaclebeing in the view of the camera; and the calibration range is measuredas the operation range for the robot corresponding to the set imagerange so that a target mark obstacle is avoided in the measuredcalibration range.
 9. A non-transitory computer-readable medium encodedwith a program for enabling a processor to execute the following steps,the processor controlling a calibration device that associates a robotcoordinate system at a robot and an image coordinate system at a cameraby placing a target mark at the robot, controlling the robot so as tomove the target mark, and detecting the target mark at multiple pointsin a view of the camera, the program being executed by the processor toperform operations comprising: setting an image range in the imagecoordinate system; and measuring a calibration range as an operationrange for the robot corresponding to the image range beforeimplementation of calibration by controlling the robot to move thetarget mark and detecting the target mark by determining whether or notthe target mark is detectable in the image range that was set in theimage coordinate system by moving the target mark and detecting thetarget mark repeatedly, thereby measuring a calibration range as anoperation range for the robot corresponding to the set image range,wherein during implementation of the calibration, the robot iscontrolled so as to move the target mark in the calibration range. 10.The non-transitory computer-readable medium encoded with a program forenabling a processor to execute steps according to claim 9, wherein theprogram is further executed by the processor so that the image range inthe image coordinate system is set so as to avoid a target mark obstaclebeing in the view of the camera; and the calibration range is measuredas the operation range for the robot corresponding to the set imagerange so that a target mark obstacle is avoided in the measuredcalibration range.
 11. A calibration device that associates a robotcoordinate system at a robot, position information in an imagecoordinate system at a first camera of a stereo camera, and positioninformation in an image coordinate system at a second camera of thestereo camera by placing a target mark at the robot, controlling therobot so as to move the target mark, and detecting the target mark atmultiple points in a view of the stereo camera including at least thefirst camera and the second camera, the calibration device comprising: amemory configured to store a program; and a processor configured toexecute the program and control the calibration device to: set a firstimage range in the image coordinate system at the first camera; set asecond image range in the image coordinate system at the second camera;measure a first calibration range as a first operation range for therobot corresponding to the first image range before the first camera andthe second camera are calibrated by controlling the robot to move thetarget mark and detecting the target mark by using the first camera bydetermining whether or not the target mark is detectable in the firstimage range that was set in the image coordinate system by moving thetarget mark and detecting the target mark repeatedly, thereby measuringa first calibration range as a first operation range for the robotcorresponding to the set first image range; and measure a secondcalibration range as a second operation range for the robotcorresponding to the second image range before the first camera and thesecond camera are calibrated by controlling the robot to move the targetmark and detecting the target mark by using the second camera bydetermining whether or not the target mark is detectable in the secondimage range that was set in the image coordinate system by moving thetarget mark and detecting the target mark repeatedly, thereby measuringa second calibration range as a second operation range for the robotcorresponding to the set second image range, wherein while the firstcamera and the second camera are calibrated, the robot is controlled soas to move the target mark in at least one of the first calibrationrange and the second calibration range, or in a range covered by boththe first calibration range and the second calibration range.
 12. Thecalibration device according to claim 11, wherein each of the firstcalibration range and the second calibration range is set on a plane.13. The calibration device according to claim 11, wherein the firstimage range in the image coordinate system is set so as to avoid atarget mark obstacle being in the view of the stereo camera and thesecond image range in the image coordinate system is set so as to avoida target mark obstacle being in the view of the stereo camera; and thefirst calibration range is measured as the first operation range for therobot corresponding to the set first image range so that a target markobstacle is avoided in the measured first calibration range and thesecond calibration range is measured as the second operation range forthe robot corresponding to the set second image range so that a targetmark obstacle is avoided in the measured second calibration range.
 14. Acalibration method executed by a calibration device including a memorystoring a program and a processor, the calibration device associating arobot coordinate system at a robot, position information in an imagecoordinate system at a first camera of a stereo camera, and positioninformation in an image coordinate system at a second camera of thestereo camera by placing a target mark at the robot, controlling therobot so as to move the target mark, and detecting the target mark atmultiple points in a view of the stereo camera including at least thefirst camera and the second camera, the processor executing the programto perform operations of the calibration method comprising: setting afirst image range in the image coordinate system at the first camera;setting a second image range in the image coordinate system at thesecond camera; measuring a first calibration range as a first operationrange for the robot corresponding to the first image range before thefirst camera and the second camera are calibrated by controlling therobot to move the target mark and detecting the target mark by using thefirst camera by determining whether or not the target mark is detectablein the first image range that was set in the image coordinate system bymoving the target mark and detecting the target mark repeatedly, therebymeasuring a first calibration range as a first operation range for therobot corresponding to the set first image range; and measuring a secondcalibration range as a second operation range for the robotcorresponding to the second image range before the first camera and thesecond camera are calibrated by controlling the robot to move the targetmark and detecting the target mark by using the second camera bydetermining whether or not the target mark is detectable in the secondimage range that was set in the image coordinate system by moving thetarget mark and detecting the target mark repeatedly, thereby measuringa second calibration range as a second operation range for the robotcorresponding to the set second image range, wherein while the firstcamera and the second camera are calibrated, the robot is controlled soas to move the target mark in at least one of the first calibrationrange and the second calibration range, or in a range covered by boththe first calibration range and the second calibration range.
 15. Thecalibration method according to claim 14, wherein the first image rangein the image coordinate system is set so as to avoid a target markobstacle being in the view of the stereo camera and the second imagerange in the image coordinate system is set so as to avoid a target markobstacle being in the view of the stereo camera; and the firstcalibration range is measured as the first operation range for the robotcorresponding to the set first image range so that a target markobstacle is avoided in the measured first calibration range and thesecond calibration range is measured as the second operation range forthe robot corresponding to the set second image range so that a targetmark obstacle is avoided in the measured second calibration range.
 16. Anon-transitory computer-readable medium encoded with a program forenabling a processor to execute the following steps, the processorcontrolling a calibration device that associates a robot coordinatesystem at a robot, position information in an image coordinate system ata first camera of a stereo camera, and position information in an imagecoordinate system at a second camera of the stereo camera by placing atarget mark at the robot, controlling the robot so as to move the targetmark, and detecting the target mark at multiple points in a view of thestereo camera including at least the first camera and the second camera,the program being executed by the processor to perform operationscomprising: setting a first image range in the image coordinate systemat the first camera; setting a second image range in the imagecoordinate system at the second camera; measuring a first calibrationrange as a first operation range for the robot corresponding to thefirst image range before the first camera and the second camera arecalibrated by controlling the robot to move the target mark anddetecting the target mark by using the first camera by determiningwhether or not the target mark is detectable in the first image rangethat was set in the image coordinate system by moving the target markand detecting the target mark repeatedly, thereby measuring a firstcalibration range as a first operation range for the robot correspondingto the set first image range; and measuring a second calibration rangeas a second operation range for the robot corresponding to the secondimage range before the first camera and the second camera are calibratedby controlling the robot to move the target mark and detecting thetarget mark by using the second camera by determining whether or not thetarget mark is detectable in the second image range that was set in theimage coordinate system by moving the target mark and detecting thetarget mark repeatedly, thereby measuring a second calibration range asa second operation range for the robot corresponding to the set secondimage range, wherein while the first camera and the second camera arecalibrated, the robot is controlled so as to move the target mark in atleast one of the first calibration range and the second calibrationrange, or in a range covered by both the first calibration range and thesecond calibration range.
 17. The non-transitory computer-readablemedium encoded with a program for enabling a processor to execute stepsaccording to claim 16, wherein the program is further executed by theprocessor so that the first image range in the image coordinate systemis set so as to avoid a target mark obstacle being in the view of thestereo camera and the second image range in the image coordinate systemis set so as to avoid a target mark obstacle being in the view of thestereo camera; and the first calibration range is measured as the firstoperation range for the robot corresponding to the set first image rangeso that a target mark obstacle is avoided in the measured firstcalibration range and the second calibration range is measured as thesecond operation range for the robot corresponding to the set secondimage range so that a target mark obstacle is avoided in the measuredsecond calibration range.